1. Field of the Invention
The present invention relates to a technique for estimating allocation information of an image sensing device and indices allocated on a physical space.
2. Description of the Related Art
As one method of calculating the position and orientation of an image sensing device that moves in a physical space, a method of calculating the position and orientation of an image sensing device based on indices allocated or set on the physical space and an image sensed by the image sensing device is known (see non-patent reference 1). With this method, the image coordinates of the indices in the image sensed by the image sensing device are detected, and the position and orientation of the image sensing device are calculated using the correspondence between the detected image coordinates of the indices and their three-dimensional (3D) coordinates.
However, the method of calculating the position and orientation of an image sensing device using only indices on the physical space can no longer calculate those of the image sensing device when the indices are occluded by obstacles or when they have poor visibility. Hence, the following robust method has been proposed (see non-patent reference 2). That is, indices are attached to the image sensing device itself as an object to be measured, and are externally sensed using a plurality of other image sensing devices set on a ceiling or the like, and measurement is made using images sensed by these image sensing devices together. Each image sensing device which externally senses an image of the object to be measured (the image sensing device in this case) will be referred to as an objective viewpoint camera hereinafter.
In the aforementioned measurement method, the position and orientation of each index (to be referred to as allocation information of each index hereinafter) on a 3D coordinate system as a reference for measurement (to be referred to as a reference coordinate system hereinafter) need be calculated in advance. The reference coordinate system is defined to have one point on the physical space as an origin, and three axes which are orthogonal to each other at this origin as X-, Y-, and Z-axes. The position and orientation of each objective viewpoint camera on the reference coordinate system, and parameters unique to the camera such as the distortion characteristics, focal length, image center, and the like need be calculated in advance. The position and orientation of each camera will be referred to as the allocation information of each camera or camera extrinsic parameters hereinafter. Also, the parameters unique to each camera such as the focal length, image center, and the like will be referred to as camera intrinsic parameters hereinafter.
An operation or process of acquiring the allocation information of each index will be referred to as calibration of each index hereinafter. The calibration of each index can be attained by a manual measurement using a ruler, protractor, surveying jig, and the like. However, the manual measurement suffers problems such as much labor and poor precision. For this reason, each index is conventionally calibrated using image information by a simple method with high precision.
The allocation information of an index whose allocation information is expressed by the position of one point on the physical space (to be referred to as a point index hereinafter) can be calculated by a bundle adjustment method (see non-patent reference 3). The bundle adjustment method simultaneously calculates the positions of points on the physical space and the position and orientation of the image sensing device upon sensing each sensed image based on a large number of images sensed from various directions. More specifically, this method optimizes the positions of points and the position and orientation of the image sensing device so as to minimize the sum of errors between the positions of points observed on the sensed image and those of the points on an image calculated from the positions of the points and the position and orientation of the image sensing device.
On the other hand, in case of an index (to be referred to as a 2D index hereinafter) whose allocation information is expressed by the position and orientation on the reference coordinate system like an index having a two-dimensional shape such as a square or the like, the aforementioned method of calculating the positions of points cannot be directly applied. Hence, non-patent reference 4 discloses a method of calculating the allocation information of the 2D index and point index by the same method as the bundle adjustment method. In order to measure the allocation information of each index on the reference coordinate system, pieces of allocation information of some indices on the reference coordinate system must be given. An index whose allocation information on the reference coordinate system is given will be referred to as a reference index hereinafter. The reference coordinate system may be defined by these reference indices. Respective indices to be calibrated must be sensed to be linked together from the reference indices.
Note that “sensed to be linked together from the reference indices” indicates, for example, the following situation. That is, the reference indices and index A are simultaneously sensed on some images, and indices A and B are simultaneously sensed on some other images. Also, index B and other indices are simultaneously sensed on still other images. In this manner, “sensed to be linked together from the reference indices” indicates the situation in which the relationships among the reference indices and other indices can be derived by following data.
On the other hand, a process of calculating the allocation information of the camera and the parameters unique to the camera such as the focal length, image center, and the like is called camera calibration (see non-patent reference 5). In general, the camera calibration is implemented by observing a plurality of points whose positions on a 3D coordinate system are known. This requires a special reference object for calibration such as an elaborated 3D reference object or the like. Also, a method of implementing calibration using a reference plane on which points whose 2D positions are known are drawn is known (see non-patent reference 6).
In the field of photogrammetry, calculating the camera intrinsic parameters is called an interior orientation, and calculating the camera extrinsic parameters is called an exterior orientation. The interior orientation may be separately done using a calibration board or the like before the exterior orientation, or may be done simultaneously with the exterior orientation. The exterior orientation is classified into a relative orientation and absolute orientation. Calculating the relative relationship among the positions and orientations of a camera sensed at a plurality of viewpoints is called a relative orientation. Calculating the position and orientation of a camera on the reference coordinate system is called an absolute orientation. The information of the position and orientation of a camera calculated by the relative orientation corresponds to a relative relationship between cameras, and represents the position and orientation of each camera on a temporary coordinate system different from the reference coordinate system.
A case will be examined below wherein a large number of objective viewpoint cameras are fixed to the ceiling of a building or the like over a broad range, and they are to be calibrated. A simplest implementation method based on the related arts is a method of individually executing the absolute orientation of each camera without any relative orientation. For this purpose, a reference object that specifies the reference coordinate system can be sensed by each individual camera.
Also, in another method, after the relative relationship between the cameras is calculated by the relative orientation, the absolute orientations of respective cameras are simultaneously determined by calculating the transformation between the temporary coordinate system of the relative orientation and the reference coordinate system. In this case, by simultaneously observing a reference object which has no relation to the relative coordinate system by a plurality of cameras, the interior orientation and relative orientation are simultaneously executed. Next, by observing another reference object that specifies the reference coordinate system by one or more cameras, the absolute orientation is executed. Note that it is a common practice to individually execute the interior orientation of each individual camera prior to setting of the cameras, and to execute the relative orientation and absolute orientation in turn after the cameras are set.
As described above, with any method described above, the absolute orientation of an objective viewpoint camera is implemented by observing the “reference object that specifies the reference coordinate system”. Note that the reference object that specifies the reference coordinate system gives a set of points (reference points) whose positions on the reference coordinate system are known. Such an object is formed by allocating, on the physical space, a plurality of indices (reference indices) each of which has known allocation information on the reference coordinate system. The precision of the absolute orientation largely depends on the distributions of the reference points in the images and space to be observed. That is, if the distribution of the reference points to be observed is sufficient, the absolute orientation with a sufficiently high precision can be executed.
[Non-Patent Reference 1]
Sato, Uchiyama, and Tamura: “Registration method in mixed reality”, Transactions of the Virtual Reality Society of Japan, vol. 8, no. 2, pp. 171-180, 2003.
[Non-Patent Reference 2]
Sato, Uchiyama, and Yamamoto: “UG+B method: A Registration Framework Using Subjective and Objective Viewpoint Cameras, and Orientation Sensor”, Transactions of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 391-400, 2005.
[Non-Patent Reference 3]
The Japan Society of Photogrammetry and Remote Sensing, “Analytic Photogrammetry, Revised edition”, 1989.
[Non-Patent Reference 4]
Kotake, Uchiyama, and Yamamoto: “A Marker Calibration Method Utilizing A Priori Knowledge on Marker Arrangement”, Transactions of the Virtual Reality Society of Japan, vol. 10, no. 3, pp. 401-410, 2005.
[Non-Patent Reference 5]
Ueshiba, Okatani, and Sato: “Camera Calibration”, IPSJ SIG Notes, 2005-CVIM-148(1).
[Non-Patent Reference 6]
Z. Zhang: “Flexible Camera Calibration by Viewing a Plane from Unknown Orientations,” Proc. 7th Int'l Conf. on Computer Vision (ICCV'99), Vol. 1, pp. 666-673, 1999.
[Non-Patent Reference 7]
Kato, M. Billinghurst, Asano, and Tachibana: “An Augmented Reality System and its Calibration based on Marker Tracking”, Transactions of the Virtual Reality Society of Japan, vol. 4, no. 4, pp. 607-616, 1999.
[Non-Patent Reference 8]
Junichi Rekimoto: “A Method of Constructing of Augmented Reality using the 2D matrix code”, Interactive System and Software V, Kindai Kagaku sha, 1997.
[Non-Patent Reference 9]
G. Klein and T. Drummond: “Robust Virtual Tracking for Non-instrumented Augmented Reality,” Proc. International Symposium on Mixed Reality and Augmented Reality (ISMAR'03), pp. 113-122, 2003.
[Non-Patent Reference 10]
Jun Sato: “Computervision—Geometry of Vision—”, Corona Publishing, Co., Ltd., 1999.
[Non-Patent Reference 11]
Tomasi, C. and Kanade, T.: “Shape and motion from image streams under orthography: A factorization method,” International Journal on Computer Vision, vol. 9, no. 2, pp. 137-154.
However, if both objective viewpoint cameras and reference indices are fixed on the physical space, it is not always easy to acquire reference points required for the absolute orientation with high precision. If a sufficient number of reference indices cannot be allocated within the image sensing range of each objective viewpoint camera, the absolute orientation of the objective viewpoint camera cannot be precisely executed. However, since the fixed objective viewpoint camera has a limited image sensing range, it can only sense images of limited reference indices. For this reason, reference indices need be additionally allocated within the range that the objective viewpoint camera can sense. However, a sufficient number of reference indices cannot often be allocated due to physical constraints. The reference points desirably have a 3D distribution. However, the reference indices can be allocated only on the floor surface and wall surface in practice, and such limited allocations of the reference indices cause a drop in the precision of the absolute orientation. It is possible to sense images of a 3D calibration jig on which reference points are allocated while moving that jig to prescribed positions. However, such an image sensing process requires a lot of labor and the allocation precision of the jig poses another problem.
Conventionally, the relative orientation and absolute orientation in the camera calibration are independent processes, and require much labor. In addition, since the camera calibration and index calibration are executed as independent processes, and require much labor.
Upon calibration of indices allocated on the physical space, if there is an index allocated at a position where that index cannot be sensed to be linked together from the reference index, the allocation information of this index on the reference coordinate system cannot be calculated. Thus, the allocation positions of indices are limited, and indices must be allocated with excessively more density.
As described above, it is required to execute the calibration of objective viewpoint cameras and indices more simply and precisely.